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This research report presents the final year results of a three-year research project on computational 
thinking (CT). The project, funded by the National Science Foundation, involved training teachers in 
grades four through six to implement Scalable Game Design and LEGO® EV3 robotics during 
afterschool clubs. Thirty teachers and 531 students took part in the Year-3 study that blended game 
design and robotics. Eight of these teachers and 98 students participated in a large urban city in 
Pennsylvania, while the remaining 22 teachers and 433 students participated in rural Wyoming. This 
paper reports on the results as it pertained to teacher outcomes, specifically, teachers’ development 
of CT beliefs and engineering practices. 
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Computer science is a rapidly growing field (Bureau of Labor Statistics, U.S. Department of 
Labor, 2016) that when coupled with engineering offers youth a broad, multidisciplinary pathway to 
occupations that provide both intrinsic and extrinsic rewards and benefits. Computing is integral to 
science, technology, engineering, and mathematics (STEM) in general and roughly 67% of 
computing jobs will be in non-STEM industries (Israel et al., 2015). Underrepresented minority 
students have limited opportunities to engage in high-quality STEM education. As access to school- 
based technology increases, opportunity expands to out-of-school time (OST) programs. Given the 
data-rich context in which we live, this expansion fills a critical need to identify the tools, pedagogy, 
and practices deemed essential for promoting STEM learning. 

This report describes our Year-3 study, which blended game design and robotics to provide 
elementary school teachers with a robust curriculum that would allow them to engage students in 
computational thinking (CT) at high levels. While several teachers were familiar with robotics, few 
had previously worked with game design. Because competency in computer science and engineering 
demands an understanding of mathematical and scientific processes and problem solving, we believe 
our pedagogical approach—to couple game design and robotics within the context of culture—is 
both necessary and innovative. Culture is defined as “a group’s individual and collective w ays of 
thinking, believing, and knowing, which includes their shared experiences, consciousness, skills, 
values, forms of expression, social institutions, and behaviors” (Tillman, 2002, p. 4) and can be used 
to motivate students to learn difficult concepts. Game design and robotics were used to promote CT 
as students engaged in abstraction, logical thinking, and debugging to create game conditions using 
simple commands and to make the robot perform a task. Specifically, we were interested in how 
teachers’ beliefs and practices changed as a result of their work in STEM with rural or urban 
elementary students during OST. 

The research questions that guided the study reported here were as follows: 

How did teachers’ beliefs about computational thinking change as a result of the study? 

How did teachers’ engineering beliefs and practices change as a result of the study? 

How did teachers’ instruction change in comparison to baseline observations? 
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How did focal teachers describe their teaching practices during OST? 


Theoretical Framework 

The theoretical framework is grounded in theories of cognitive development that emphasize a 
developmental and constructivist view of the relationship between cognition and culture. The Saxe 
(1999) model is grounded in developmental and constructivist theories of learning and emphasizes 
the relationship between cognition and culture. Saxe argued that individuals create new knowledge 
while participating in culturally influenced goal-structured activities that occur in social settings. The 
model focuses on three areas: (a) goals for learning structured by common cultural practices, (b) 
particular cognitive forms and functions created to reach goals, and (c) identifiable characteristics 
involved in the interplay across learning in different cultural contexts. 

First, the goal structure of cultural practice consists of the tasks or activities that must be carried 
out. “Games are inherently artifacts of culture through which cultural roles, values, and knowledge 
bases are transmitted” (Nasir, 2005, p. 6). In African American communities, for example, goal 
structures may be communal as success of the group is valued over individual success (Coleman et 
al., 2016). Intricate roles of help-seeking and help-offering strategies occur during game design, 
revealing the intertwining characteristics of individual and sociocultural systems of cognitive 
processing (Nasir, 2005). These roles may also be evident in game design and robotics, particularly if 
teams of students work together. In game design, goal structures are associated with creating unique 
agents and developing functional games to maximize points; in robotics, goals are associated with 
movement and carrying out specific tasks. 

Second, according to the Saxe model, sign forms—such as counting systems and cultural 
artifacts — are needed to execute and influence goals that emerge in cultural practice. “Practice- 
linked goals are influenced by many dynamics of activity, including social interaction between those 
engaged in a practice, the organizational structure of a practice, individuals’ prior goals and 
understandings, and artifacts, norms, and conventions of the practice” (Nasir, 2002, p. 216). There 
are shifting cognitive processes in game design to adapt the game to meet gamers’ needs and goals 
(Nasir, 2005). These cognitive processes may be noticeable in game design and robotics as students 
developed CT strategies via learning progressions. The role of the teacher as a facilitator is critical to 
students’ development of these skills. 


Literature Review 
While STEM encompasses a wide range of disciplines, the literature review focuses on digital 
game design, robotics, and computational thinking as teachers used CT strategies and engineering 
practices to broaden equitable participation in STEM. 


Digital Game Design 

K-12 students are part of a digital and gaming culture. Game design and simulation have been 
used to address elementary and middle school students’ motivation and interest in computer science 
courses and careers (Webb et al., 2012). Software tools like Scratch (Mouza et al., 2016; Israel et al. 
2015) and Scalable Game Design (SGD) (Repenning et al., 2015) have been used successfully to 
help children design digital games. SGD uses instructional units to support game design and 
simulations through the use of AgentSheets and AgentCubes, allowing students to engage in higher- 
level thinking skills as they use code to move the agent through obstacles in a game (Repenning et 
al., 2015). Scratch, AgentSheets, and AgentCubes were used in this study to provide teachers with 
robust curriculum. 
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Robotics 

Sullivan and Heffernan (2016) conducted a review of research on robotics construction kits 
(RCKs) as computational manipulatives in P-12 settings. They define computational manipulatives 
as those that have internal computing capabilities, programming, or microcomputers embedded in the 
hardware. Grounded in the LOGO (i.e., logic oriented graphic oriented) programming language, 
computational manipulatives allow children to engage in analytical and embodied cognition (Papert, 
1993). Sullivan and Heffernan assert that RCKs have a dual function as students engage in both 
cognitive and physical aspects of learning. For example, during LEGO® robotics, students may 
mimic the physical motions of the robot as it travels along a path while also engaging in reasoning, 
reflection, discussion, and problem solving to complete a robotics task. Most importantly, students 
may follow a learning progression, such as sequencing, causal inference, conditional reasoning, and 
systems thinking, as they interact with RCKs (Sullivan & Heffernan, 2016). 


Computational Thinking 

CT is an evolving field that also emerged from the work of Papert (1993), who was the first to 
use the term. Wing (2006) defines CT as a human endeavor that “involves solving problems, 
designing systems, and understanding human behavior by drawing on the concepts fundamental to 
computer science” (p. 33). CT is a cognitive skill that all students are expected to use across 
disciplines and in multiple contexts (Mouza et al., 2016; Weintrop et al., 2016; Yadav et al., 2014). 
Weintrop et al. (2016) developed a taxonomy to incorporate the computational nature of mathematics 
and science into more recent educational endeavors. Their taxonomy for CT in mathematics and 
science includes the following practices: using data, modeling and simulation, computational 
problem solving, and systematic thinking. We use this definition of CT because it aligns well with 
both the computer science and engineering aspects of our study. 


Methodology 


Participants and Setting 

Thirty teachers participated in most aspects of the Year-3 study. However, we limit this report to 
19 teachers new to the project and 4 teachers, who chose to continue from the Year-2 study; these 23 
teachers fully participated in the Year-3 study. 

All of the teachers received the same training, which consisted of two logistics meetings and an 
eight-week course on game design and robotics. Game design was taught by a computer scientist, 
who was a member of the research team. Lesson plans focused primarily on creating mazes, Frogger, 
and PacMan games using SGD. Robotics was taught by a science educator, who was also a member 
of the research team. Lesson plans focused on using LEGO® EV3 robotics kits to make the basic car, 
gyro boy, and sumo bot. MINDSTORMS® programming controlled the robot’s movements and use 
of ultrasonic, color, and touch sensors. 


Data Analyses and Data Sources 

Mixed methods were used to analyze data in the Year-3 study. Quantitative data were used to 
examine changes in teachers’ CT beliefs (Yadav et al., 2014) and engineering practices (i.e., 
Engineering Education Beliefs and Expectations Instrument for Teachers (EEBEI-T) (Nathan et al., 
2010). Internal reliability of the survey instruments revealed Cronbach alphas were in the acceptable 
range (Black, 1999): CT survey (a = 0.76); EEBEI-T (a = 0.70). The CT survey consists of 21 items 
related to understanding CT (i.e., definition), dispositions (i.e., comfort, interest, and classroom 
practices), and future careers. Examples of these items included: “Computational thinking involves 
thinking logically to solve problems; Knowledge of computing will allow me to get a better job. 
Using a 4-point Likert scale, scores ranged from | (strongly disagree) to 4 (strongly agree). Two 
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constructs on the EEBEI-T were analyzed for this report: STEM Integration of Content and Applied 
Knowledge (13 items) and Engineering Cycle (5 items) to query teachers about the use of curriculum 
and engineering practices. Since these two constructs were specifically developed for use with 
teachers in Project Lead the Way (PLTW), reliability parameters are not available. However, the 
EEBEI-T survey in its entirety and inclusive of these two constructs does possess established content 
validity (Nathan et al., 2011). An example of these items included: “J use curriculum activities that 
rely on application and design activities as a way to introduce students to basic laws in math and 
science.” Using a 5-point Likert scale, scores ranged from (0-never; 1-almost never; 2-sometimes; 3- 
often; and 4-almost always). The 7-statistic was used to analyze pre-post scores on each of these 
surveys. We set the confidence interval at 0.90 since the sample size was small (Quinn & Keough, 
2002). 

Qualitative data included using the Dimensions of Success (DoS) instrument to collect field notes 
and rate teachers’ practices (Noam et al., 2014). Factor analysis revealed the 12 DoS dimensions can 
be aggregated into two groups: student learning and learning environment (Gitomer, 2014). Student 
learning may be further divided into three domains: (a) Activity Engagement; (b) STEM & 
Knowledge Practices; and (c) Youth Development. A score of 3 constitutes what researchers 
document as reasonable evidence while a score of 4 constitutes compelling evidence. Ratings were 
shared with teachers at the end of the study year as member checks. Additionally, three teachers 
agreed to serve as focal participants to share their experiences and reflections on the project. 


Results 


Computational Thinking (CT) and EEBEI-T Surveys 

Twenty-three teachers completed pre-post CT surveys. The data (see Table 1) reveal significant 
differences on the CT survey from pre-post: (¢ = -2.173; p = 0.041; Cohen’s d= 0.34). Cohen’s d 
shows a small effect size for this increase. Two teachers who completed the CT did not complete the 
EEBEI-T. Table 1 also reveals significant differences from pre-post on the modified EEBEI-T (¢ = - 
2.882; p = 0.009; Cohen’s d= 0.78). Cohen’s d shows a large effect size for this increase. 
Interpretation of the data reveal teachers ranged from ‘almost never’ to ‘sometimes’ on the STEM 
Integration of Content and Applied Knowledge and Engineering Cycle constructs on the pre-survey 
but tended toward ‘sometimes’ on these constructs on the post-survey. Thus, teachers made progress 
on use of engineering practices in the Year-3 study. 


Table 1: Results of Teachers’ Survey 


Construct Pre-Survey Standard Post-Survey Standard 
Mean Deviation Deviation 
CT Survey (n=23) 3.34 0.30 3.45* 0.30 
EEBEI (n=21) 1.88 0.57 2.40* 0.77 


* p <0.05 


Dimensions of Success 

We observed 23 teachers during the Year-3 study using the DoS instrument. However, three 
teachers were observed only once and, therefore, removed from the sample. Mean ratings (M/) of the 
first observation were used as a baseline, and mean ratings (M2) of the last observation were used to 
show growth. The interval between teaching episodes was typically four to six weeks unless the 
teacher participated in multiple years. Four teachers engaged in co-teaching and 12 taught lessons 
individually. Thus, 16 total observations were analyzed by lesson type to show trends on the three 
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domains of interest (see Table 2). Teachers’ ratings were slightly higher on robotics than game 
design at the baseline. Teachers’ ratings showed compelling evidence (M>= 4.0) for Purposeful 
Activities, STEM Engagement, STEM Content Learning, and Relationships on final observations in 
the game design context. Scores increased from M;= 2.9 to M2 = 3.6 for game design. However, the 
lowest rating was on the Relevance dimension (M>= 2.8). DoS scores dropped from M = 3.3 to Mz= 
3.2 in the robotics context, with the most substantial drop on STEM & Knowledge Practices. Overall, 
teachers’ practice improved from M, = 3.1 to M2 = 3.3 regardless of context. The strongest domain at 
the end of the study was Activity Engagement, which was followed by Youth Development and 
STEM & Knowledge Practices. Relevance (M> = 2.4) was the weakest dimension in Year 3 as well as 
in prior years (Leonard et al., under review). 


Table 2: Analysis of DoS Observation 


ACTIVITY ENGAGEMENT STEM & KNOWLEDGE PRACTICES YOUTH DEVELOPMENT 
Observation Student Purposeful STEM STEM Inquiry Reflection Relationship Relevance Student Mean 
Participation Activities Engagement Content Voice 
Learning 
Baseline — 2S) 3) 3.6 2.8 28) 2.4 3.8 1.8 3.0 2.9 
Gaming 
(n=9) 
Final — 3.7 4.0 4.0 4.0 3.2 3.2 4.0 2.8 3.3 3.6 
Gaming 
(n =6) 
Baseline — Syl Rall 3.9) 3.4 33) 3.0 3.9 2.4 Bell 3.3 
Robotics 
(n=7) 
Final — 3.3 3.4 3.7 3.0 3.1 2.8 3.8 2.2 3.1 3.2 
Robotics 
(n= 10) 
Total 3.0 3.4 Soll Ball Shall Dell 3.8 el Ball 3:1 
Baseline 
(n=16) 
Total Final 3.4 3.6 3.8 3.4 3.1 2.9 3.9 2.4 3.2 3.3 
(n=16) 


Case Narratives 

The cases of three focal teachers allowed us to examine the complexity of enabling CT and 
engineering practices during OST. Each of the teachers worked with students during before- or 
afterschool clubs teaching game design and robotics for 20 hours in each context for a total of 40 
hours or more. These teachers — one White female (rural), one African American female (urban), 
and one White male (urban) — wrote extensive narratives about their teaching experiences during 
this project. Pseudonyms were used for anonymity. 

Annette. A technology specialist in rural Wyoming, Annette worked with nine White fourth- and 
fifth-grade students for two hours per day, four days per week, before and after school. Class sizes in 
rural Wyoming are typically 10-14 students. As a facilitator, Annette allowed her students to develop 
agency by having “deeper experiences, more meaningful conversations with peers, and ownership of 
the project” while “working towards a common goal.” During game design, her students faced a 
challenge creating PacMan games. “The students understood that the chasers needed to chase their 
main character, but were not quite sure how to accomplish this. Now, I was able to teach them about 
hill climbing and diffusion because it was finally relevant. The main character had to emit something 
that would attract the chaser...[like] ‘stinky feet.’ Tracking the scent, is called a hill climb, and the 
scent being detected [is] diffusion, not to be confused with osmosis, which requires water. The video 
game needed to be programmed to take the main character’s scent and diffuse it throughout the 
game. The chaser needed to be able to ‘sniff in all four directions and start to move in the direction 
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that the scent was strongest. I got lots of nods, and ‘Oh, that makes sense.’” Thus, Annette provided 
students with opportunities to use more advanced CT strategies (Repenning et al., 2015) within a 
constructivist paradigm, which aligned with Saxe’s (1999) cognitive theory and supported the finding 
for high Activity Engagement and STEM & Knowledge Practices on the DoS. Her ratings across the 
three DoS domains were stable VM; = 3.6 (game design) to M> = 3.6 (game design). 

Charles. As a teacher-leader in science and technology at an urban elementary school in 
Pennsylvania, Charles worked with 20 predominantly African American students for 90 minutes two 
times per week. None of his students had participated in robotics or game design before. He 
described how he laid the foundation for learning: “...I explained that computers are machines that 
are programmed to perform computations or actions based on the input they receive. We discussed 
the inputs that the human body might receive through senses, as well as the concept of involuntary 
reflexes.” Then he distributed cards with conditional statements to help students understand how 
commands are implemented. “For example, if they heard someone say ‘thank you,’ someone else 
would respond, ‘you’re welcome.’ I explained that computer programming was binary ‘if/then’ or 
‘yes/no’ language that we needed to understand before creating a program.” This activity revealed 
that cognitive and physical activity may be intertwined in computer science as well as robotics 
(Sullivan & Heffernan, 2016). However, Charles found that challenges to programming “involved 
rudimentary debugging. Having learned from their own mistakes, students became more effective in 
identifying problems in their classmates’ games. Students were able to create increasingly intricate 
programs. Some chose to create multiple levels, multiple games, or multiple rules on a single 
worksheet.” These activities showed how cognitive processes shifted as students engaged in social 
interactions (Nasir, 2005) and communal practices (Coleman et al., 2016). It also supported 
reasonable evidence of STEM Knowledge & Practices and Youth Development. Charles’ ratings 
across the three DoS domains increased from M; = 3.0 (game design) to M2 = 3.4 (game design). 

L’wanda. As the school librarian, L’ wanda was able to work with 20 students for 90 minutes a 
week at two different urban schools in Pennsylvania. None of the 40 predominantly African 
American students had participated in robotics before, and only one had previously experienced 
game design. L’ Wanda was able to use “‘iPads to reinforce the skills students needed to develop CT 
skills. There were 10 robotics kits so students were able to work in groups of two. However, some 
students preferred to work in groups of three or four. Students used iPads to access templates on the 
LEGO website that helped them learn to build the robots. Students developed their research skills as 
they searched for information to improve their designs. Students used the scientific method to 
develop programs that would allow their robots to go longer distances and avoid obstacles. As they 
raced their robots against other teams, they constantly revised their designs and made changes to the 
programs to build better bots that would help them achieve their goals. Participation in the club was a 
turning point that changed attitudes toward learning. My students were able to see themselves as 
capable, intelligent leaders, who were able to be producers and not just consumers. Students used 
iPads to create videos that explained the steps they took to create their robots. These videos 
demonstrated the depth of their understanding of the material and showed that they had a real sense 
of purpose.” Thus, L’ Wanda’s students engaged in learning progressions as they used RCKs to 
develop CT. L’ Wanda’s ratings across the three DoS domains increased from M, = 3.0 (robotics) to 
M> = 3.2 (robotics). 


Discussion 
The findings of our Year-3 study are promising. Teachers’ pre-post surveys on computational 
thinking beliefs and engineering practices improved significantly, and overall, teachers’ ability to 
implement STEM during OST showed improvement when baseline data were compared to final 
observations. Trends in terms of the overall DoS show teachers met the threshold of 3 on all domains 
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during the last observation. The narratives were rich and provided examples of the learning that 
occurred during OST. Teachers were facilitators that provided guided inquiry when necessary to help 
students engage in independent learning. Both game design and robotics provided students with 
opportunities to engage in highly cognitive tasks without sacrificing cultural norms (Nasir, 2005). 
For example, students engaged in communal learning in urban settings (Coleman et al., 2016). 
Teachers encouraged student collaboration, and some children became peer coaches at times. 
Children were successful in designing digital games and performing robotics challenges (Leonard et 
al., 2016; Repenning et al., 2015; Sullivan & Heffernan, 2016). Nevertheless, some teachers did not 
promote cultural relevance or career awareness during their lessons. Although we encouraged 
culturally relevant pedagogy (CRP) (Ladson-Billings, 2009), some teachers paid cursory attention to 
students’ culture and were less likely to link computer science and engineering tasks to STEM 
careers. We will address the issue of CRP in future studies through teacher reflection, case studies, 
and co-generative dialogue. 


Significance 

In this study, robotics and game design were used to broaden STEM participation in rural and 
urban communities. We learned that game design facilitated co-generative dialogue that allowed 
learners to take familiar concepts and apply them to a range of complex tasks including the creation 
of representations and models. We strongly believed that the research design not only facilitated CT 
applicable to STEM but also promoted the kinds of social engagement and collaboration that will 
build 21° century skills and normative habits that allow students to develop computer science and 
ICT skills. 


Limitations 

The results of this study are limited to the participants and settings where the study took place 
and should not be generalized to teachers in other contexts. One limitation was the smaller number of 
robotics clubs for baseline observations, which may have skewed the DoS tabulations. Another 
limitation associated with the quantitative data is the absence of reliability parameters for the two 
EEBEI-T constructs used in this study. 
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